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“Debugging is twice as hard as writing the 
code in the first place. Therefore, if you write 
the code as cleverly as possible, you are, by 
definition, not smart enough to debug it.” 


— Brian W. Kernighan © 2020 Philip Koopman 1 
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__ System Level Testing Sc 
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e Excessive defect “escapes” 
to field testing 
e Majority of testing effort is go et aera pee aE ct lb ciny Pil 
ad hoc exploratory testing be: ee ae ) 
e Acceptance testing is the = . 
: The Famous “First Bug 
only testing done on system (But, Edison invented the term https://goo.gl/cVLpcx) 
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= System test is last line of defense against shipping bugs 
e System-level “acceptance test’ emphasizes customer-type usage 
e Software test emphasizes aspects not visible to customer 
— E.g., is the watchdog timer turned on and working? Ss aatietn: outers 


= System test plan covers all requirements 
e Every product requirement is tested 


e Non-customer-visible requirements are tested 


e Need to deal with embedded system I/O 


= Each bug found in system test is a huge deal 


Effective System Testing Mellon 


University 





—- Ad hoc testing helps, but should not be primary method 


- Especially non-functional requirements 


— Use a HAL and swap in a test simulator harness 


You should find few (<5%?) bugs in system test 
Bug found in system test is a process failure 

— System requirement defects should be most of what you find 
— Make sure “one-off” bugs aren't just tip of the iceberg process problems 
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https://goo.gl/uWXQBC 
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Product Testing Wont Find All Bugs Mellon 
= Testing bad software = One third of faults take more 
simply makes it less bad than 5000 years to manifest 
e Testing cannot produce good —_pretuctisir soul of fesereh and Ceveopment 
software aI on its own 28(1), p. 2-14, 1984. (Table 2, pg. 9, 60 kmonth column) 


e Do you test for more than 
5000 years of use? 





e Your customers will regularly 
experience bugs that you will 
not see during testing 
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INTERNATIONAL DATE LINE 
-11 


F-22 Raptor Date Line Incident 


m February 2007; six aircraft fly to Japan 
e $360 million per aircraft 
e Computer crash crossing the International Date Line:,- 
— No navigation, communications, fuel management. ... 
— Escorted to Hawaii by tankers % 
— Could have lost all six if poor visibility 
m Cause: 
e “It was a computer glitch in the millions of lines 
of code, somebody made an error in a couple lines of | 
the code and everything goes.” [hitps://goo.gl/edGdAL] 
= Related: F-16 inverts when crossing equator 
e Found in simulation. (Perhaps an urban legend?) 


e But still should put designers on notice of such bugs 
http://catless.ncl.ac.uk/Risks/3.44.html#subj1.1 
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Bug Farms: Concentrations of Buggy Code ee Dis 

= 90/10 rule applied to bug farms: : 
e 90% of the bugs are in 10% of the modules 
e Those are the most complex modules 

= Bug farms can be more than just bad code 
e Bad design that makes it tough to write code 
e Too complex to understand and test 
e Poorly defined, confusing interfaces 

= Fixing bug farms: 
e Refactor the module, redesign the interface 
e Often, smart to throw away and redesign that piece 
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Top 10 Risks of Poor Embedded Software Quality pol 
10. Your module fails unit test (Tie with #9) 
9. A bug is found in peer review (Tie with #10) 
8. The system fails integration or software testing 
7. The system fails acceptance testing 
6. You get a field problem report | 
5. Your boss wakes you up at 2 AM because a Big Customer is off-line 
4. You get an airplane ticket to a war zone to install a software update 
3. You hear about the bug on social media 
2. Your corporate lawyers ask you to testify in the lawsuits 
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And, the Number One Worst Way To Find A Bug: 
1. The reporters camped outside your house ask you to comment on it 
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System Test Best Practices Nilo 

= Test all system requirements 

e Everything its supposed to do 

e Fault management responses 

e Performance, extra-functional reqts. 
= Acceptance test vs. software test a, ee 

e Acceptance test is from customer point of view — domain testing 

e SW test uses internal test interfaces — software testing skills 
= System test pitfalls: 

e Impractical to get high coverage; won't find all bugs 

e Testing fault management is hard if you haven't planned for it 
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[http://www.groundhog.org] 
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Weather Capital of the World. 
= The one test every embedded system must pass is the dreaded 


Groundhog Test. To perform this test: 

e Connect a prototype unit to a de-energized power supply 

e Stand back, look the other direction and throw the switch. 

e If you see your shadow, that means 6 more weeks of development 





— Rick Miu 


“T SPEND A LOT OF TIME ON THIS TASK. 
T SHOULD WRITE A PROGRAM AUTOMATING IT!" 


WRITING 
CODE.“ 
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HOW LONG CAN YOU WORK ON MAKING A ROUTINE. TASK MORE 
EFFICIENT BEFORE YOU'RE SPENDING MORE TIME THAN YOU SAVE? 
(ACROSS FIVE YEARS) 


—— HOW OFTEN You DO THE. TASK ——__ 
/rev Oppy DAILY WEEKLY MONTHLY YEARLY 





https://xkcd.com/1205/ 


